Should this value be capped when using unified cache to avoid exceeding the model context length? I think it could be set to min(n_ctx_train, n_ctx), or add a parameter to allow the user to change it.

I guess we can cap it to n_ctx_train. The only use case for n_ctx > n_ctx_train that comes to mind is self-extend, but lately this technique seems less relevant.

We can also cap it for the non-unified case?

We can also cap it for the non-unified case?

What would happen to the leftover slots? I may be misunderstanding the way split cache works, but my assumption would be that these slots would never be used, and it would be wasted memory. So if that's capped, it should be done at context creation.

Right, we should do the capping at context creation in the llama_context constructor. Currently we have some additional logic for this in llama-model:

Since we no longer need the padding logic (as of #16148 and related) we should simplify this.

I'll push a separate PR for this and then will come back to polishing this one.

This is now rebased on top of the changes in #16812. The result is that we determine the KV cache size during context creation and there should be no leftover KV cells.

Note that since we now cap the context size to the training context size, the user code is recommended to query llama_n_ctx and llama_n_ctx_seq after creating the llama_context in order to obtain the actual context size. I'll add comments in llama.h to reflect this.

Will try to clean-up this PR next and will open it for review when ready.

This branch should not be reached due to the capping above on line 117. But keeping it in case the capping logic gets changed in the future.

I am not completely convinced about this, I think it may create confusion, and add complexity to applications. The server and other applications using the unified cache need a sequence length limit independent of n_ctx, but that should probably be a different parameter that defaults to min(n_ctx, n_ctx_train). This would be an application parameter, not part of the llama.cpp API.

Sounds good. Just remove the capping here?

Yes, I think it would be preferable to not have a limit here. The user should be able to override the model n_ctx_train, and it is easier to do it this way than with a KV override.

Moved the capping logic to the llama-server.

Is there a reason why this can't be default params in arg.h?

I'll see if I can make it the default - I thought that some of the examples might not like it.

Hmm yeah I didn't notice that there are multiple example all using n_parallel

In this case, maybe we can use a dedicated variable for server, like params.n_parallel_server ?

This can be useful when auto-generating the documentation for server args

this warning should be moved inside the if condition above, right?

Also maybe I forgot this from a discussion before, but currently in which case we need to retry with a small batch size?

The main case for retrying with smaller batches was back when we didn't have ggml_set_rows and we always had to search for contiguous set of cells (KV slots) inside the cache buffer to place the input batch. Now with ggml_set_rows this is no longer needed and technically, retrying with a smaller batch size almost has almost no purpose except in some rare cases.

But generally, when llama_decode returns 1, you should retry with a smaller batch.

Maybe an error could be returned here if n_ctx is not a multiple of n_seq_max, since that's likely to be a mistake.

I added a warning. The problem that I see with throwing an error is that the user might often want to use the default training context for example split among 3 sequences. And in the majority of cases the training context typically a power of 2 would not be divisible by 3, resulting in an error.